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Abstract 

If  a  loss  function  is  available  specifying  the  social  cost  of  an  error 
of  measurement  in  the  score  on  a  unidimensional  test,  an  asymptotic  method, 
based  on  item  response  theory,  is  developed  for  optimal  test  design  for  a 
specified  target  population  of  examinees.  Since  in  the  real  world 
such  loss  functions  are  not  available,  it  is  more  useful  to  reverse 
this  process;  thus  a  method  is  developed  for  finding  the  loss  function 
for  which  a  given  test  is  an  optimally  designed  test  for  the  target 
population.  An  illustrative  application  is  presented  for  one  operational 
test. 
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Estimating  the  Imputed  Social  Cost  of  Errors  of  Measurement* 

For  a  unidimensional  test,  the  error  of  measurement  is  the  dif¬ 
ference  between  the  examinee's  true  ability  e  and  the  estimate  of  this 
ability  represented  by  the  examinee's  test  score  6  .  Since  discrepancies 
between  6  and  6  may  lead  to  erroneous  decisions  about  the  examinee 
(misclassif ication,  erroneous  acceptance  or  rejection),  there  is  an 
expected  social  cost  associated  with  any  pair  of  values  (6,6)  •  This 
cost  is  given  by  some  loss  function  L(6,e)  . 

The  obvious  problem,  here  called  Problem  1,  is:  Given  the  loss  func¬ 
tion  L(0,6)  ,  how  can  we  build  an  (optimal)  n  -item  test  that  will 
minimize  the  expected  loss  over  a  specified  target  population  of  examinees, 
subject  to  certain  constraints  on  the  statistical  characteristics  of  the 
items  in  the  available  item  pool?  Using  item  response  theory,  [Lord, 

1980;  Hulin,  Drasgow,  and  Parsons,  1983],  a  solution  of  this  problem 
will  be  given  here  for  a  unidimensional  test. 

Unfortunately,  in  practice  it  is  unlikely  that  L(6,0)  will  be 
known  to  the  test  designer.  Something  of  practical  value  can  still  be  salvaged, 
however,  if  we  can  deal  with  Problem  2:  Given  an  existing  unidimensional 
test  and  a  specified  target  population  of  examinees,  find  the  loss  func¬ 
tion  L(9,0)  for  which  this  test  is  an  optimally  designed  test.  If  the 

*The  theoretical  work  in  Sections  1-4  was  supported  by  contract 
N00014-80-C-0402,  project  designation  NR  150-453  between  the  Office  of 
Naval  Research  and  Educational  Testing  Service.  The  empirical  work, 
using  ETS  data,  was  supported  by  ETS  funds.  The  writer  is  very  much 
indebted  to  Martha  L.  Stocking,  who  was  responsible  for  obtaining  the 
empirical  results  reported  in  Section  5. 
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loss  function  found  for  Problem  2  does  not  agree  with  our  intuitive 
notions  as  to  what  is  appropriate,  we  will  probably  redesign  future 
test  forms  to  avoid  this  discrepancy. 

In  order  to  solve  Problem  2,  it  is  necessary  first  to  solve  Problem  1; 
this  is  done  in  the  first  section.  The  solution  to  Problem  2  is  outlined 
in  the  second  section.  Invariance  under  transformations  of  the  ability 
scale  is  discussed  in  Section  3.  In  Section  4,  a  method  for  estimating 
the  ability  distribution  of  the  target  population  is  discussed. 

An  illustrative  application  to  an  actual  test  is  given  in  Section  5.  The 
final  section  briefly  discusses  some  implications  for  optimal  test 
design. 

It  is  assumed  here  that  all  item  parameters  have  been  determined 
by  pretesting  to  sufficient  accuracy  so  that  they  can  be  treated  as 
known.  The  illustrative  example  and  some  of  the  discussion  are  based  on 

the  three-parameter  logistic  model  of  the  item  response  function  (with 
which  the  reader  is  assumed  to  be  familiar),  but  the  proofs  of  the  main 
results  are  much  more  general.  The  examinee's  actual  score  6  is 
assumed  to  be  the  maximum  likelihood  estimate  of  3  ,  calculated  from 
the  examinee's  responses  to  the  n  test  items. 

1.  Minimizing  Expected  Loss 

For  a  group  of  examinees  at  a  given  ability  level  3  ,  the  conditional 


expected  loss  is  by  definition 
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S(L|e)  -  /  L(e,e)  <j>(e|e)  de 


where  <J>(0|e)  is  the  conditional  distribution  of  0  and  6  denotes  expecta¬ 
tion.  If  the  distribution  of  ability  0  in  the  target  population  is  denoted 
by  g(0)  ,  then  the  overall  (unconditional)  expected  loss  is  by  definition 


6  (L)  -  f  S  (L  |  0)  g(0)  d0 


This  is  the  quantity  to  be  minimized  by  optimal  test  design. 


Loss  Function 

Certain  reasonable  assumptions  will  be  made  about  the  loss  function: 

1.  L(0 ,0 )  *  0  (because  when  0  »  0  ,  there  is  no  error  of  measure¬ 
ment  and  hence  no  loss  due  to  error  of  measurement) . 

2.  When  0*0,  L(0,0)  >  0  . 

3.  When  6  is  near  0  ,  the  loss  function  and  its  first  two 
derivatives  with  respect  to  0  are  continuous,  the  third 
derivative  is  bounded.  [These  conditions  will  guarantee  the 
convergence  of  (3) . ] 

4.  The  loss  function  does  not  change  too  sharply  with  changes  in 
0  (as  will  be  discussed  later) . 

For  fixed  0  ,  expand  L(0,0)  in  powers  of  0  -  0  ,  obtaining 


L(0 ,0)  -  L(0,0)  +  (0  -  0)L’ (0,0)  +  \  (0  -  0)2L"(0,0)  +  ... 
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where  L'(0,6)  and  L"(0,0)  denote  successive  derivatives  of  L(0,0) 
with  respect  to  0  ,  evaluated  at  0=0.  The  first  term  vanishes 
because  there  is  no  error  of  measurement  when  0=0.  The  second 
term  vanishes  because  for  fixed  0  ,  L(0,0)  has  a  minimum  at  9=0. 

Consequently; 

L(0  ,9 )  =  y  (9  -  0)^  L" (0,0)  plus  higher  order  terms.  (3) 

Higher  powers  of  (0-9)  can  be  neglected  if  n  is  not  too  small, 
since  9  -*•  0  in  probability  as  n  -*•  00  [Lord,  1980,  p.  59]. 

When  (3)  is  substituted  into  (1),  L"(0,0)  comes  out  from  under  the 
integration  sign.  It  is  then  apparent  that  asymptotically  (that  is, 
for  large  n  ) 

6  (L|0)  =  L"(0,0)  Var(0|0)  .  (4) 

In  item  response  theory,  the  asymptotic  (conditional)  variance  of  0 
is  the  reciprocal  of  the  test  information  function  1(0)  [Lord,  1980, 
Section  5.3].  Thus  we  shall  rewrite  the  expected  loss  (2)  as 


S(L) 


L" (6,6)  g (9 ) 
1(0) 


d0 


(5) 


Information  Function 

The  item  response  function  =  P^(0)  is  the  probability  of  a 
correct  response  to  item  i  by  a  randomly  chosen  examinee  at  ability 


level  0  .  The  information  function  is 
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1(6) 


1 

Var(§ | 0) 


n 

I 

i-1 


>.2 


PiQi 


(6) 


where  =  1  -  and  =  dP^/d0  . 

Ordinarily  P^  depends  on  an  item  difficulty  parameter  . 
Furthermore,  b^  is  typically  simply  a  translation  parameter:  it 
affects  P^^  only  through  the  difference  0  -  bi  .  In  this  standard 
situation,  b^  also  affects  P^  only  through  the  difference  0  -  b.^  . 
Thus  the  area  under  any  function  F  of  P^  and  P^  over  the  whole 
range  of  0 


00  oo 

I  F (0  -  b±)  d0  -  /  F(0)  de 

— oo  —a? 


is  independent  of  b^  .  The  area  under  the  test  information  function 
thus  does  not  depend  on  b^  in  these  typical  models,  which  will  be 
assumed  here. 

In  the  special  case  where  P^Q)  is  the  three-parameter  logistic 
function 


1  ”  ci 

Pi(e)  "  Ci+  1  +  exp[-1.7ai(e  -  b  )] 

I 


(7) 


we  have 
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and 


/  i(e)  d e  =  / 


pi2 

Errde 

i  iwi 


E 

i 


Pi 

Vi 


d_Ii 

de 


de 


1 

E  / 
i  cJ 


p; 

r5"dPi 

ivi 


1.7a  1  c 

'  (i  -  ?;  >  "i 

1  Ci  1 


1  -  c , 


1.7a 
1  -  c 


i 

i 


ci  log  Pi] 


n  1.7a 

=  1  - —  (1  -  c.  +  c  log  c  )  .  (8) 

i-1  1  "  ci  1  1  i 

This  area  does  not  depend  on  . 

Test  Constraints 

There  are  always  constraints  on  the  availability  of  items  for  test 
construction.  Item  writers  can  control  to  a  considerable  extent  the 
difficulty  level  of  the  items  they  write.  The  discriminating  power  of 
the  available  items,  however,  can  ordinarily  be  increased  only  by  writing 
more  items  and  then  discarding  a  larger  percentage  of  the  items  written — 
an  expensive  procedure. 

It  will  be  assumed  here  that  the  test  developer  has  available  an 
unlimited  pool  of  items  at  whatever  difficulty  levels  he  or  she  may  specify. 
The  items  in  the  pool  have  already  been  pretested;  faulty  items, 
especially  those  with  low  discriminating  power,  have  already  been  dis¬ 
carded.  The  test  developer  is  to  build  parallel  forms  of  a  test  from 
the  item  pool,  selecting  items  only  on  the  basis  of  their  difficulty  b  , 


Imputed  Social  Cost 
8 


so  that  each  parallel  form  has  the  same  distribution  of  .  Items 
cannot  be  selected  on  the  basis  of  their  discriminating  power,  since  all 
items  not  discarded  after  pretesting  must  eventually  be  used.  In  the  actual 
test  produced,  the  frequency  distribution  of  other  item  parameters,  such  as 
item  discriminating  power,  is  to  be  the  same  as  in  the  total  pool  of  pre¬ 
tested  items.  It  will  be  assumed  here  that  in  the  item  pool  the  distrlbc  >n 
of  other  item  parameters  is  independent  of  the  item  difficulty  b^  .  Thi 
assumption  should  be  checked  empirically  for  any  practical  application. 

This  assumption  may  fail  to  hold  because  of  the  essential  nature  of 
the  test  items;  often  it  also  fails  to  hold  simply  because  pretest  item- 
test  biserials  have  been  used  instead  of  the  IRT  discrimination  parameter 
a^  to  exclude  poorly  discriminating  items  from  the  available  item  pool. 

When  item-test  biserials  are  used  in  this  way  for  multiple-choice  items, 
the  harder  the  item,  the  higher  the  a 1  parameter  must  be  for  the  item 
to  escape  exclusion  from  the  item  pool.  This  is  true  because  among  items 
with  identical  a^  ,  the  more  guessing  the  lower  the  item-test  biserial. 

It  follows  from  these  assumptions  that  the  total  area  under  the 
test  information  function  is  fixed.  The  task  of  the  test  developer  is  to 
minimize  £(L)  by  choice  of  b^  (  i  =  1,2, . . . ,n  ) ;  no  other  relevant 
variables  are  available  to  the  test  developer  for  achieving  this  minimization. 

Minimization 

By  the  Cauchy  inequality, 

/^•/IM  /  •/L"g  )2  . 
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Here,  the  first  integral  is  twice  the  expected  loss  (5)  written  in 
abbreviated  notation.  Transposing,  we  have 

/  ^  >  (  /  v^g  )2  1  /  I  .  (9) 

In  Problem  1,  L"(9,0)  and  g(6)  are  known;  furthermore  /  1(0)  d0 

— oo 

is  fixed  by  the  reasoning  of  the  last  two  subsections.  It  follows  that  if 
there  is  an  1(9)  such  that  equality  holds  in  (9),  then  this  is  the 
1(9)  that  minimizes  the  expected  loss  (5).  Equality  will  hold  in  (9) 
provided 


1(0)  is  proportional  to  /L"(0,0)  g(0) 

Monetary  Units 

The  loss  function  L(0,0)  is  necessarily  expressed  in  terms  of  some 
arbitrary  unit  (dollar,  peso,  ...).  It  may  be  convenient  to  choose  this 
unit  so  that  the  area  under  /[L"^,©)  g(0)]  is  equal  to  /  1(9)  d0  , 

this  last  being  a  known  and  fixed  quantity  determined  by  n  and  by  the 
item  parameters,  excluding  the  b^  ,  of  the  item  poc/i.  Once  this  choice 
of  unit  has  been  made,  the  expected  loss  will  be  minimized  if  the  test 
developer  can  build  a  test  with 
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Building  the  Test 

Birnbaum  [1968,  Section  20.6]  suggested  an  effective  cut-and-try 

method  for  building  a  test  having  (approximately)  a  prespecified 

'target'  information  function.  The  method  is  outlined  in  Lord  [1980, 

Section  5.4].  The  method  follows  easily  from  the  fact  that  the  test 

information  function  is  simply  a  sum  of  the  information  functions 

(P'3p.Q.)  of  the  items  included  in  the  test, 
l  ii 

The  method  is  effective  provided  the  target  information  curve  is 
not  too  irregular  and  does  not  vary  too  rapidly  as  a  function  of  6  . 

The  results  obtained  here  hold  under  this  condition.  If  the  target  curve 
is  too  irregular,  it  will  not  be  possible  to  build  a  test  having  the  desired 
information  function  by  selecting  items  on  bj^  from  the  available  item  pool. 

Practical  Procedure  (Summary) 

A 

Given  L(0,0)  and  g(0)  ,  to  build  k  parallel  test  forms  of  length 
n  that  approximately  minimize  the  expected  loss: 

1.  Plot  a^  and  c^  against  b^  to  verify  that  the  distribution 
of  a^  and  c^  in  the  item  pool  is  approximately  the  same 

at  all  levels  of  b^  ,  as  assumed  in  the  subsection  titled 
Test  Constraints. 

2.  Compute 


/  1(9)  d0  = 


5  *  7  p±  d9 

i=l_oo 


where  M  is  the  number  of  items  in  a  large  item  pool.  Note  that 


this  integral  does  not  depend  on  the  distribution  of  b^  in  the  pool. 
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3.  Choose  monetary  units  so  that 


oo 

f  /[l" (0,0)  g(e) ]  de 

— .oo 


is  equal  to 


k  /  1(0)  d0 

— OO 

for  some  integer  k  . 

4.  Selecting  items  only  on  their  ,  use  Birnbaum's  method 

to  select  a  pool  of  nk  items  such  that  the  sum  of  the  nk 
item  information  functions  is  approximately  equal  to 
/[L"(0,0)  g(0)]  . 

5.  Divide  the  nk  selected  items  into  k  test  forms  of  n  items 
each,  all  approximately  parallel  to  each  other. 

2.  The  Loss  Function  for  Which  a  Given  Test  Is  an  Optimal  Test 

If  a  given  test  is  an  optimal  test,  then  (10)  holds  and 


L"(0,0) 


=  i2(e) 

g(0) 


(i 


Consequently,  the  loss  function  is  given  approximately  by  (3)  and  (11) 


T  V  A 


^  1 


I2(6) 


( A  -  AV 


r\ 
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For  fixed  0  ,  this  is  the  equation  of  a  parabola.  When  n  is  not  too 
small,  0  will  be  close  to  0  and  (12)  will  provide  an  adequate 
approximation  to  the  lose  function  for  those  values  of  >  that  are  likely 
to  be  observed.  For  0  close  to  0  ,  the  desired  loss  function  can  be 
computed  from  (12)  for  anv  given  test,  provided  g(  •)  is  snecified. 

3.  Transformation  of  the  Score  Scale 


Loss  functions  have  an  invariance  property  that  is  important  in 
dealing  with  problems  of  test  design.  Consider  the  social  cost  in 
dollars  of  an  error  of  measurement  at  a  given  ability  level.  If  the 
error  of  measurement  (the  discrepancy  between  the  actual  test  score  and 
the  true  ability  of  which  it  is  an  estimate)  is  specified  as  a  multiple 
of  its  standard  error,  asymptotically  (for  large  n  )  the  loss  in 
dollars  will  be  the  same  no  matter  what  scale  is  used  for  measuring 
ability. 

Instead  of  using  the  0  scale  of  ability,  suppose  we  use  the 
number-right  true-score  scale,  given  by  the  monotonic  continuous 
transformation 

n 

C  -  z  p  .  (e )  .  (13) 

i=l  1 

The  examinee's  obtained  score  should  now  be  taken  to  be 

n 

£  =  Z  P.(0) 
i=l  1 


j 

-1 


(14) 
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(Note  that  we  need  to  use  here  the  maximum  likelihood  estimator  of  £ 
defined  by  (14),  not  the  examinee’s  number  of  right  answers.)  If 
0  differs  from  8  by  K  times  S . E . (0 | 0 )  ,  then,  asymptotically, 

£  will  differ  from  £  by  K  times  S.E.(£|£)  .  Asymptotically, 

K[ S . E . (€ ) 6 ) ]  is  actually  the  same  error  of  measurement  on  the  0  scale 
as  K[S.E. (£ | £) ]  is  on  the  £  scale;  thus  the  social  cdnsequences  of 
this  error  will  be  the  same  regardless  of  the  scale  used. 

Let  0(£)  denote  the  inverse  of  transformation  (13).  Expressed  on 
the  £  scale,  the  loss  function  (12)  becomes 


L^ (£  ,£)  =  \ 


g[e  (£)  ] 


[e(b  -  e(£)]2 


where  g(  )  and  I(  )  denote  the  same  functions  as  previously.  This 
equation  could  be  used  as  it  stands,  but  for  reasons  of  symmetry,  it  may 
be  preferable  to  expand  it  for  fixed  £  in  powers  of  £  -  £  .  The  result 
is  found  to  be 


L  a  r)  sIOiiiU 

L£U’U  2  gte  (£)  ] 


12  a  -  o2  . 


(15) 


Equation  (15)  is  used  here  to  represent  the  loss  function  when  the 
obtained  score  is  £  rather  than  0  .  This  transformation  has  an 
advantage  for  presenting  experimental  results,  since  the  number-right 


score  scale  is  more  familiar  to  us  than  the  0  scale. 
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Note  again  that  the  actual  monetary  loss  is  the  same  regardless  of 
the  scale  against  which  it  is  plotted.  This  invariance  makes  the  loss 
function  much  more  useful  for  guiding  test  design  than  the  information 
function.  Expressed  on  the  9  scale,  the  test  information  function  for 
9  is  typically  a  bell-shaped  curve;  expressed  on  the  £  scale,  the 
test  information  function  for  £  is  necessarily  a  U  -shaped  curve 
[Lord,  1980,  Chapter  6].  This  lack  of  invariance  makes  it  difficult  to 
use  the  test  information  function  as  a  convincing  basis  for  test  design. 

A.  Estimating  the  True  g(0) 

By  its  definition,  expected  loss  (2)  requires  specification  of 
the  distribution  9  in  the  target  population.  It  is  important  to  note 
that  the  distribution  of  9  in  the  target  population  is  not  an  adequare 
estimate  of  g(9)  ,  the  true  distribution  of  9  .  The  reason  is  that 
9  contains  errors  of  measurement  and  thus  has  a  larger  variance  than 
9  .  Since  g(9)  appears  in  the  denominator  of  (12),  it  is  particularly 

important  to  estimate  g(9)  as  accurately  as  possible. 

To  obtain  the  numerical  results  of  Section  5,  the  true  distribution 
of  number-right  true  score  (13),  here  denoted  by  f(£)  ,  was  estimated  by 
Method  20  [Lord,  1980,  Chapter  16],  Since  £  is  necessarily  >_  £jP^(-°°) 
an  estimated  lower  limit  for  £  was  set  at  E^c^  ,  where  ^  represents 
the  estimated  c  parameter  of  item  i  .  For  purposes  of  Section  5  all 
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item  parameters  were  estimated  under  the  three-parameter  logistic  model 
(7)  by  the  computer  program  LOGIST  [Note  1]. 

The  required  estimate  of  g(e)  for  the  target  population  was 
obtained  from  the  Method  20  estimate  of  f(^)  by  the  relation 

g(6)  -  f (5)  f§  •  (16) 

The  derivative  in  (16)  is  the  derivative  of  (13) ,  estimated  in  practice 
by  computing  E^Pj^Q)  from  estimated  item  parameters. 

5.  Illustrative  Example 

A  representative  sample  of  19,949  examinees  tested  in  1981-82  was 
obtained  for  the  Sentence  Sense  test  in  Form  3EJP  of  the  New  Jersey  College 
Basic  Skills  Placement  Test.  This  competency  test  consists  of  35  four- 
choice  items  requiring  the  examinee  to  distinguish  correct  from  incorrect 
English  expression.  The  test  is  used  primarily  to  assign  certain  entering 
college  students  to  remedial  English  classes. 

The  item  responses  of  all  19,949  examinees  were  analyzed,  using  LOGIST 
to  estimate  the  item  parameters  of  all  items  in  the  test.  The  true  distri¬ 
bution  of  9  for  the  target  population  was  estimated  as  described  in 
Section  4  (for  this  purpose,  a  response  chosen  at  random  from  the  four 
choices  was  supplied  wherever  an  examinee  failed  to  respond  to  an  item). 
The  test  information  function  (6)  was  calculated  from  the  estimated  item 
parameters.  Finally,  the  loss  function  for  which  the  test  is  an  optimally 
designed  test  was  estimated  by  (12). 
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Figure  1  shows  the  actual  distribution  of  number-right  scores  (fre¬ 
quency  polygon) ,  the  number-right  true-score  distribution  estimated  by 
Method  20  (solid  curve),  and  the  corresponding  fitted  distribution  of 
(observed)  number-right  scores  (dotted  curve) .  The  modal  score  is  31 
right  answers  out  of  a  possible  35.  The  chi  square  between  observed  and 
fitted  number-right  score  distributions  is  at  the  86th  percentile  of  the 
chi  square  distribution  with  18  degrees  of  freedom.  In  view  of  the  large 
sample  size  (  N  =  19,949  ),  this  seems  an  adequate  fit,  as  in  indeed  sug¬ 
gested  visually  by  the  agreement  shown  in  Figure  1. 

The  estimated  loss  function  (12)  for  which  the  test  is  an  optimally  de- 
signed  test  is  plotted  in  Figure  2  against  the  B  and  0  Scales.  The  direc 
tion  of  the  6  scale  is  reversed  from  the  conventional  direction  in  order  to 
improve  visibility.  Loss  is  shown  on  the  vertical  scale.  In  this  and 

A 

the  next  figure,  the  parabola  for  any  given  6  is  drawn  only  for  0  values 
within  two  standard  errors  of  the  true  0  . 

The  figure  shows  that  the  Sentence  Sense  test  is  built  as  if  it  were 
important  to  measure  accurately  at  high  ability  levels  as  well  as  at  low 
ability  levels.  Clearly,  this  is  not  appropriate  for  a  competency  test — 
the  test  should  assign  high  losses  to  errors  of  measurement  at  low  ability 
levels  but  not  at  high  ability  levels.  The  more  difficult  items  in  the 
test  should  be  replaced  by  easier  items. 

The  estimated  loss  function  (15)  for  which  the  test  is  an  optimally 
designed  test  is  plotted  in  Figure  3  against  number-right  true  score  and 


Figure  3.  Estimated  Loss  Function  for  NJBSCPT,  3EJP,  Sentence  Sense,  N  =  19949 
as  a  function  of  true  score  (  £  )  and  estimated  true  score  (  C  ) • 
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estimated  number-right  true  score.  For  ease  of  viewing,  both  scales  at 
the  bottom  of  the  figure  run  in  the  opposite  direction  from  the  scales 
at  the  bottom  of  Figure  2.  This  plot  is  easier  to  interpret  than  Figure  2 
since  we  are  more  accustomed  to  the  number-right  score  scale  than  to  the 
0  scale.  The  plot  looks  very  different  from  Figure  2  because 

1.  The  loss  function  for  a  number-right  score  of  34  is  not  shown. 

The  loss  function  for  this  score  is  rather  high  and  would 
obscure  too  much  of  the  rest  of  the  figure. 

2.  A  wide  range  at  the  high  end  of  the  6  scale  is  compressed 
into  a  small  range  of  number-right  scores,  as  shown  in  the 
following  table: 


C  : 

8  10 

12 

14 

16 

18 

20  22 

24  26  28  30  32 

34 

0  : 

-5.2  -3.1 

-2.4 

-1.9 

-1.6 

-1.3 

-1.0  -.7 

-.4  -.2  .1  .5  1.0 

2.1 

Again,  it  appears  that  the  test  discriminates  at  high  true  score  levels, 
where  discrimination  is  not  really  desired.  The  loss  function  at  £  =  34 
(not  plotted)  shows  a  loss  of  approximately  100  when  £  is  two  standard 
errors  from  £  .  For  number-right  scores  of  30  and  below,  the  shape  of 
the  loss  function  seems  very  appropriate  for  a  competency  test,  with  very 
high  losses  attributed  to  errors  of  measurement  at  low  score  levels. 
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6.  Discussion 

In  the  case  of  a  minimum  competency  test,  the  social  losses  arising 
from  errors  of  measurement  will  be  high  for  examinees  near  the  cutting 
score,  which  is  always  near  the  low  end  of  the  score  scale.  Social  losses 
will  be  near  zero  for  examinees  far  from  the  cutting  score,  since  decisions 
about  these  examinees  will  not  be  changed  by  small  errors  in  their  scores. 

For  a  college  admissions  test,  it  would  seem  reasonable  to  expect 
that  errors  of  measurement  in  the  scores  of  high  ability  sutdents  will 
result  in  relatively  high  social  losses.  Somewhat  lower  social  losses 
should  be  expected  to  result  from  errors  in  the  scores  of  low  ability 
students. 

In  the  case  of  grade  school  tests  of  'ability'  or  of  vocabulary,  it 
has  sometimes  been  argued  that,  to  be  fair,  the  standard  error  of 
measurement  of  the  test  score  should  be  roughly  the  same  for  each 
individual  (see,  for  example,  Hulin  et  al.,  1983,  p.  90).  The  first 
difficulty  with  this  approach  is  that  its  implications  for  test  design 
when  the  test  score  is  0  are  completely  different  than  when  the  test 
score  is  Z,  or  simply  the  number  of  right  answers.  Although  equality  of 
standard  errors  of  measurement  at  all  ability  levels  has  strong  intuitive 
appeal,  there  is  no  clear  way  to  decide  whether  this  equality  should  hold 
on  the  0  scale,  or  on  the  number-right  score  scale,  or  on  some  other 
scale.  It  cannot  hold  simultaneously  on  two  different  scales  unless 
one  scale  is  a  linear  transformation  of  the  other. 
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In  any  case,  any  goal  of  equal  standard  errors  of  measurement  at 
different  ability  levels  is  completely  incompatible  with  the  goal  of 
minimizing  expected  social  loss  due  to  errors  of  measurement.  If  we 
wish  to  minimize  social  loss,  we  must,  other  things  being  equal, 
mobilize  our  test  development  resources  so  as  to  measure  most  accurately 
at  those  ability  levels  where  the  most  people  are  found.  We  cannot 
waste  items  in  order  to  secure  accurate  measurement  at  ability  levels 
where  only  a  few  people  will  be  affected,  unless,  of  course,  there  is 
a  very  high  loss  function  at  these  ability  levels.  In  a  word,  accuracy 
of  measurement  in  sparsely  populated  stretches  of  the  ability  range  must 
be  sacrificed,  other  things  being  equal,  in  order  to  obtain  more 
accurate  measurement  in  heavilv  nopulated  stretches. 

As  a  concrete  example,  consider  a  vocabulary  test  for  grade  5  and 
suppose  our  test  is  built  to  minimize  overall  expected  loss.  Suppose  also, 
as  might  be  reasonable  for  such  a  test,  that  the  expected  loss  at  a  fixed 
ability  level  is  constant  across  ability  levels,  so  that,  by  (12), 

S[L(9,e)  le]  .ii^p.S[(6  .  e)2|ej  .  I  ifiSi  ^  .  K 

where  K  is  some  constant.  It  follows  that 

Var(e|e)  "  TW  "  Hiw 
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Since  g (0 )  is  small  for  extreme  6  ,  the  standard  error  of  measurement, 
/Var(0|0)  ,  will  in  this  case  be  very  much  larger  for  examinees  with 
extreme  0  than  for  examinees  with  moderate  0  .  Thus  in  this  case  the 
goal  of  equal  standard  errors  of  measurement  at  all  ability  levels  is 
utterly  incompatible  with  minimizing  overall  expected  loss.  This  is 
simply  an  illustration  of  the  fact  that  if  we  wish  to  minimize  overall 
expected  loss, our  measurement  effort  must  be  concentrated  on  the  sub¬ 
ranges  of  ability  that  are  most  highly  populated  in  the  target  population. 

To  summarize,  in  respect  to  a  unidimensional  test: 

1.  Given  the  loss  function,  the  distribution  of  ability  in  a 
target  population,  and  certain  constraints  on  the  available  item  pool,  a 
method  has  been  described  for  designing  a  test  that  will  minimize  expected 
loss. 

2.  Given  a  test  and  also  the  distribution  of  ability  in  the 
target  population,  a  method  has  been  described  for  finding  the  loss 
function  for  which  this  test  is  an  optimally  designed  test  given  certain 
constraints  on  the  available  item  pool. 

3.  Minimizing  social  loss  is  in  general  incompatible  with 
equal  measurement  accuracy  across  examinees.  To  minimize  social 
loss,  measurement  accuracy  must  be  high  (other  things  being  equal) 
over  ability  ranges  that  are  heavily  populated,  and  relatively  low 
over  ranges  that  are  sparsely  populated. 
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AFIIRL/MPD 

Brooks  Air  Force  Base,  TX  78235 

1  Technical  Documents  Center 

Air  Force  Human  Resources  Laboratory 
WPAFB,  OH  45433 

1  U.S.  Air  Force  Office  of 
Scientific  Research 
Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Air  University  Library 
AUL/LSE  76/443 
Maxwell  AFB,  AL  36112 

1  Dr.  Earl  A.  Alluisi 
HQ,  AFHRL  (AFSC) 

Brooks  Air  Force  Base,  TX  78235 

1  Mr.  Raymond  E.  Christal 
AFHRL /M0E 

Brooks  AFB,  TX  78235 

1  Dr.  Alfred  R.  Fregly 
AF0SR/NL 

Bolling  AFB,  DC  20332 


Air  Force 


Department  of  Defense 


1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

1  Dr.  T.  M.  Longridge 
AFHRL/OTE 

Williams  AFB,  AZ  85224 

1  Dr.  Roger  Pennell 

Air  Force  Human  Resources  Laboratory 
Lowry  AFB,  CO  80230 

1  Dr.  Malcolm  Ree 
AFHRL/MF 

Brooks  Air  Force  Base,  TX  78235 

1  LT  Tallarigo 
3700  TCHTW/TTGHR 
Sheppard  AFB,  TX  76311 

1  Dr.  Joseph  Yasatuke 
AFhRL/LRT 

Lowry  AFB,  CO  80230 


Department  of  Defense 


12  Defense  Technical  Information  Center 
Attn:  TC 

Cameron  Station,  Building  5 
Alexandria,  VA  22314 


1  Dr.  Craig  1.  Fields 

Advanced  Research  Projects  Agency 
1400  Wilson  Blvd. 

Arlington,  VA  22209 

1  Dr.  William  Graham 
Testing  Directorate 
MEPCOM/MEPCT-P 
Ft.  Sheridan,  1L  60037 


1  Mr.  Jerry  Lehnus 
HQ  MEPC0M 
Attn:  MEPCT-P 
Ft.  Sheridan,  IL  60037 

1  Military  Assistant  for  Training 
and  Personnel  Technology 
Office  of  the  Under  Secretary  of 

Defense  for  Research  and  Engineering 
Room  3D129,  The  Pentagon 
Washington,  DC  20301 

1  Dr.  Wayne  Sellman 

Office  of  the  Assistant  Secretary 
of  Defense  (MRA&L) 

2B269  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 

Civilian  Agencies 


1  Dr.  Patricia  A.  Butler 
NIE-BRN  Bldg. ,  Stop  #7 
1200  19th  Street,  NW 
Washington,  DC  20208 

1  Dr.  Susan  Chipman 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Arthur  Helmed 
724  Brown 

U.S.  Department  of  Education 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 

Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Washington,  DC  20550 


Civilian  Agencies 


Private  Sector 


1  Dr.  Vem  W.  Urry 

Personnel  R  &  D  Center 
Office  of  Personnel  Management 
1900  E  Street,  NW 
Washington,  DC  20415 

1  Mr.  Thomas  A.  Warm 

U.S.  Coast  Guard  Institute 
P.0.  Substation  18 
Oklahoma  City,  OK  73169 

1  Dr.  Frank  Withrow 

U.S.  Office  of  Education 
400  Maryland  Avenue,  SW 
Washington,  DC  20202 

1  Dr.  Joseph  L.  Young,  Director 
Memory  and  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


Private  Sector 


1  Dr.  James  Algina 

University  of  Florida 
Gainesville,  FL  32611 

1  Dr.  Patricia  Baggett 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  Isaac  Bejar 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  Menucha  Birenbaum 
School  of  Education 
Tel  Aviv  University 
Tel  Aviv,  Ramat  Aviv  69978 
ISRAEL 


1  Dr.  R.  Darrell  Bock 

Department  of  Education 
University  of  Chicago 
Chicago,  IL  60637 

1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.0.  Box  168 

Iowa  City,  IA  52243 

1  Dr.  Glenn  Bryan 
6208  Poe  Road 
Bethesda,  MD  20817 

1  Dr.  Ernest  R.  Cadotte 
307  Stokely 

University  of  Tennessee 
Knoxville,  TN  37916 

1  Dr.  Pat  Carpenter 

Department  of  Psychology 
Carnegi e-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  John  B.  Carroll 
409  Elliott  Road 
Chapel  Hill,  NC  27514 

1  Dr.  Norman  Cliff 

Department  of  Psychology 
University  of  Southern  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  Allan  M.  Collins 

Bolt,  Beranek,  and  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Lynn  A.  Cooper 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 
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Private  Sector 


1  Dr.  Hans  Crombag 

Education  Research  Center 
University  of  Leyden 
Boerhaavelaan  2 
2334  EN  Leyden 
THE  NETHERLANDS 

1  Dr.  Dattpradad  Divgi 
Syracuse  University 
Department  of  Psychology 
Syracuse,  NY  33210 

1  Dr.  Susan  Embertson 
Psychology  Department 
University  of  Kansas 
Lawrence,  KS  66045 

1  ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Bethesda,  MD  20014 

1  Dr.  Benjamin  A.  Fairbank,  Jr. 
McFann-Gray  and  Associates,  Inc. 
5825  Callaghan 
Suite  225 

San  Antonio,  TX  78228 

1  Dr.  Leonard  Feldt 

Lindquist  Center  for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Prof.  Donald  Fitzgerald 
University  of  New  England 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

1  Dr.  Dexter  Fletcher 

WICAT  Research  Institute 
1875  S.  State  Street 
Orem,  UT  22333 

1  Dr.  John  R.  Frederiksen 
Bolt,  Beranek,  and  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 


Private  Sector 


1  Dr.  Janice  Gifford 

University  of  Massachusetts 
School  of  Education 
Amherst,  MA  01002 

1  Dr.  Robert  Glaser 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Bert  Green 

Department  of  Psychology 
Johns  Hopkins  University 
Charles  and  34th  Streets 
Baltimore,  MD  21218 

1  Dr.  Ron  Hambleton 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01002 

1  Dr.  Paul  Horst 
677  G  Street,  //184 
Chula  Vista,  CA  90010 

1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  Jack  Hunter 

2122  Coolidge  Street 
Lansing,  MI  48906 

1  Dr.  Huynh  Huynh 

College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Dr.  Douglas  H.  Jones 
10  Trafalgar  Court 
Lawrenceville,  NJ  08648 
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Private  Sector 


Private  Sector 


i  Prof.  John  A.  Keats 

Department  of  Psychology 
University  of  Newcastle 
Newcastle,  New  South  Wales  2308 
AUSTRALIA 

1  Dr.  William  Koch 

University  of  Texas-Austin 
Measurement  and  Evaluation  Center 
Austin,  TX  76703 

1  Dr.  Fat  Langley 

The  Robotics  Institute 
Carnegi e-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 

Learning  R  &  D  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

1  Lr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Building 
University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Charles  Lewis 

Faculteit  Sociale  Wetenschappen 
Rijksuniversiteit  Croningen 
Oude  Boteringestraat  23 
9712GC  Groningen 
NETHERLANDS 


1  Dr.  Robert  Lockman 

Center  for  Naval  Analysis 
200  North  Beauregard  Street 
Alexandria,  VA  22311 

1  Dr.  Frederic  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  James  Lumsden 

Department  of  Psychology 
University  of  Western  Australia 
Nedlands,  Western  Australia  6009 
AUSTRALIA 

1  Dr.  Gary  Marco 
Stop  31-E 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Notre  Dame 
Notre  Dame,  IN  46556 

1  Dr.  Samuel  T.  Mayo 

Loyola  University  of  Chicago 
820  North  Michigan  Avenue 
Chicago,  IL  60611 

1  Mr.  Robert  McKinley 

American  College  Testing  Programs 

P.0.  Box  168 

Iowa  City,  IA  52243 


1  Dr.  Robert  Linn  1  Dr.  Robert  Mislevy 

College  of  Education  711  Illinois  Street 

University  of  Illinois  Geneva,  IL  60134 

Urbana,  IL  61801 

1  Dr.  Allen  Munro 

1  Mr.  Phillip  Livingston  Behavioral  Technology  Laboratories 

Systems  and  Applied  Sciences  Corporation  1845  Elena  Avenue,  Fourth  Floor 
66111  Kenilworth  Avenue  Redondo  Beach,  CA  90277 

Riverdale,  MD  20840 
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Private  Sector 


1  Dr.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Oklahoma  City,  OK  73069 

1  Dr.  Donald  A.  Norman 
Cognitive  Science,  C-015 
University  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Melvin  R.  Novick 

356  Lindquist  Center  for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  James  Olson 
WICAT,  Inc. 

1875  S.  State  Street 
Orem,  UT  84057 

1  Dr.  Wayne  M.  Fatience 

American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 

1  Dr.  James  A.  Paulson 

Portland  State  University 
F.O.  Box  751 
Portland,  OR  97207 

1  Dr.  James  W.  Pellegrino 
Univeristy  of  California, 

Santa  Barbara 
Department  of  Psychology 
Santa  Barbara,  CA  93106 

1  Dr.  Mark  D.  Reckase 
ACT 

P.0.  Box  168 

Iowa  City,  IA  52243 

1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15261 


Private  Sector 


1  Dr.  Thomas  Reynolds 

University  of  Texas,  Dallas 
Marketing  Department 
P.0.  Box  688 
Richardson,  TX  75080 

1  Dr.  Andrew  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.,  NVI 
Washington,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
Murray  Hill,  NJ  07974 

1  Dr.  Lawrence  Rudner 
403  Elm  Avenue 
Takoma  Park,  MD  20012 

1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Prof.  Fumiko  Samejitna 
Department  of  Psychology 
University  of  Tennessee 
Knoxville,  TN  37916 

1  Dr.  Walter  Schneider 
Psychology  Department 
603  E.  Daniel 
Champaign,  IL  61820 

1  Dr.  Lowell  Schoer 

Psychological  and  Quantitative 
Foundations 
College  of  Education 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Robert  J.  Seidel 

Instructional  Technology  Group 
HUMRRO 

300  N.  Washington  Street 
Alexandria,  VA  22314 
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Private  Sector 


1  Dr.  Kazuo  Shigemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kavauchl,  Sendai  980 
JAPAN 

1  Dr.  Edwin  Shirkey 

Department  of  Psychology 
University  of  Central  Florida 
Orlando,  FL  32816 

1  Dr.  William  Sims 

Center  for  Naval  Analysis 
200  North  Beauregard  Street 
Alexandria,  VA  22311 

1  Dr.  h.  Wallace  Sinaiko 
Program  Director 

Manpower  Research  and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  22314 

1  Dr.  Richard  Snow 
School  of  Education 


Private  Sector 


1  Dr.  William  Stout 

University  of  Illinois 
Department  of  Mathematics 
Urbana,  IL  61801 

1  Dr.  Patrick  Suppes 

Institute  for  Mathematical  Studies 
in  the  Social  Sciences 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Hariharan  Swaminathan 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massacuusetts 
Amherst,  MA  01003 

1  Dr.  Kikumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 


Stanford  University 
Stanford,  CA  94305 

1  Dr.  Kathryn  T.  Spoehr 
Psychology  Department 
Brown  University 
Providence,  RI  02912 

1  Dr.  Robert  Sternberg 
Department  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  haven,  CT  06520 

1  Dr.  Peter  Stoloff 

Center  for  Naval  Analysis 
200  North  Beauregard  Street 
Alexandria,  VA  22311 


1  Dr.  Maurice  Tatsuoka 
220  Education  Building 
1310  S.  Sixth  Street 
Champaign,  II  61820 

1  Dr.  David  Thissen 

Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

1  Dr.  Douglas  Towne 

University  of  Southern  California 
Behavioral  Technology  Labs 
1845  S.  Elena  Avenue 
Redondo  Beach,  CA  90277 

1  Dr.  Robert  Tsutakawa 
Department  of  Statistics 
University  of  Missouri 
Columbia,  M0  65201 
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Private  Sector 


1  Dr.  V.  R.  R.  Uppuluri 

Union  Carbide  Corporation 
Nuclear  Division 
P.0.  Dox  Y 

Oak  Ridge,  TN  37830 


1  Dr.  David  Vale 

Assessment  Systems  Corporation 
2233  University  Avenue 
Suite  310 

St.  Paul,  MN  55114 

1  Dr.  Kurt  Van  Lehn 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  94304 


Dr.  Howard  Wainer 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Michael  T.  Waller 
Department  of  Educational  Psychology 
University  of  Wisconsin 
Milwaukee,  WI  53201 

Dr.  Brian  Waters 
HUKRRO 

300  North  Washington 
Alexandria,  VA  22314 


Dr.  Phyllis  Weaver 
2979  Alexis  Drive 
Palo  Alto,  CA  94304 


Dr.  David  J .  Weiss 
N660  Elliott  Kail 
University  of  Minnesota 
75  East  River  Road 
Minneapolis,  MN  55455 


*  « 
& 


Dr.  Keith  T.  Wescourt 
Perceptronics,  Inc. 
545  Hiddlef ield  Road 
Suite  140 

Menlo  Park,  CA  94025 


Private  Sector 


1  Dr.  Rand  R..  Wilcox 

University  of  Southern  California 
Department  of  Psychology 
Los  Angeles,  CA  90007 

1  Dr.  Wolfgang  Wildgrube 
Streitkraef teamt 
Box  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 

1  Dr.  Bruce  Williams 

Department  of  Educational  Psychology 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Wendy  Yen 
CTB/McCraw-Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 


